INTPYTHON-751 Make query generation omit $expr unless required #396

WaVEV · 2025-09-09T17:15:54Z

In this PR a unified approach for generating MQL from Django expressions was implemented. The core idea is to centralize the control flow in a base_expression method, which decides whether the expression can be translated into a direct field: value match (index-friendly) or must fall back to $expr. This keeps the logic for wrapping and dispatching in one place, while each lookup/function only defines its own expression-building logic.

This approach also allows mixing direct field: value matches with $expr clauses within the same $match. As a result, multiple $expr entries may coexist alongside index-optimized conditions, depending on the shape of the query.

Most lookups now follow this pattern by simply implementing as_mql_expr (and optionally as_mql_path when a match-based translation is possible). Only a few special cases like Col, Func operators (except the KeyTransform) , and many more, override the base behavior directly. This structure also leaves room for future optimizations (e.g. constant folding) without having to change the overall flow.

Additionally, since MongoDB 6 does not allow nesting $expr inside another $expr, the flow in base_expression ensures that such cases are flattened. In practice, expressions are generated without redundant wrapping, so the final MQL never contains $expr within $expr.

WaVEV · 2025-09-26T03:40:00Z

django_mongodb_backend/functions.py

-    Substr.as_mql = substr
-    Trim.as_mql = trim("trim")
-    TruncBase.as_mql = trunc
+    Cast.as_mql_expr = cast


All the function does not support as_mql_path. It could be added latter if we try to simplify constants expressions

django_mongodb_backend/expressions/builtins.py

django_mongodb_backend/fields/json.py

django_mongodb_backend/base.py

django_mongodb_backend/aggregates.py

django_mongodb_backend/base.py

django_mongodb_backend/functions.py

django_mongodb_backend/query.py

django_mongodb_backend/query_utils.py

tests/model_fields_/test_embedded_model.py

tests/model_fields_/test_embedded_model_array.py

django_mongodb_backend/base.py

Jibola

Left my first round of comments. I'm going to review the tests as a second phase of the PR.

django_mongodb_backend/base.py

django_mongodb_backend/aggregates.py

django_mongodb_backend/base.py

Jibola · 2025-10-06T14:55:39Z

django_mongodb_backend/expressions/search.py

+    def as_mql_expr(self, compiler, connection):
+        lhs_mql = process_lhs(self, compiler, connection, as_path=False)
+        value = process_rhs(self, compiler, connection, as_path=False)
        return {"$gte": [lhs_mql, value]}

+    def as_mql_path(self, compiler, connection):
+        lhs_mql = process_lhs(self, compiler, connection, as_path=True)
+        value = process_rhs(self, compiler, connection, as_path=True)
+        return {lhs_mql: {"$gte": value}}
+


Why is there a $gte query in a search.text lookup?

To convert a score function into a filter I decided to express the following proposition: score_func(...) > 0.

Jibola · 2025-10-06T19:36:33Z

django_mongodb_backend/fields/embedded_model_array.py

+        return self.is_simple_column
+
+    @cached_property
+    def is_simple_column(self):


I see this validation for embedded models used in multiple places. Can you consolidate this to the query_utils file?

🤔 will try. The overall structure of the function is similar, but the type varies. I could create a meta-function that generates the appropriate function for a given type.

I didn't get better. 😬 . I will let as it is. there is some details like:
previous._field.column, previous.key_name when extract the field_name or the path. Then are different from EMFA and EMF the return, one has to validate the inner transform while the other doesn't

django_mongodb_backend/compiler.py

django_mongodb_backend/query_utils.py

Jibola · 2025-10-06T21:47:23Z

django_mongodb_backend/query_utils.py

+
+
+def valid_path_key_name(key_name):
+    return bool(re.fullmatch(r"[A-Za-z0-9_]+", key_name))


https://www.mongodb.com/docs/manual/core/dot-dollar-considerations/

Values like hashtags are also valid for path names and don't require $expr. To my knowledge so long as it's not (.) or ($) it's good.

Thanks, will adjust the expression.

what if there is some emoji or some non ascii character? 🤔

Jibola · 2025-10-06T22:28:04Z

django_mongodb_backend/fields/embedded_model.py

+    def as_mql_expr(self, compiler, connection):
+        columns, parent_field = self._get_target_path()
+        mql = parent_field.as_mql(compiler, connection)
+        for key in columns:
+            mql = {"$getField": {"input": mql, "field": key}}


potentially out of scope:
https://github.com/mongodb/django-mongodb-backend/pull/392/files#diff-0a6ce30a131a00fa88086c4c4d0d6e6232845fd11ef2bc67891fdf92e10c3743R18-R45

Is it possible to still remove $getField in as_mql_expr or is it expected that routing to as_mql_expr for embedded model queries is because of needing a getField call?

Jibola · 2025-10-06T22:31:21Z

django_mongodb_backend/lookups.py

+@property
+def can_use_path(self):
+    simple_column = getattr(self.lhs, "is_simple_column", False)
+    constant_value = is_constant_value(self.rhs)
+    return simple_column and constant_value


django_mongodb_backend/base.py

Jibola · 2025-10-06T23:41:13Z

tests/model_fields_/test_arrayfield.py

+        self.assertAggregateQuery(
+            query,
+            "model_fields__nullableintegerarraymodel",
+            [{"$match": {"field": {"$in": ([1], [2])}}}],


Does $in now expect a tuple?

Yes, but it wasn’t a new convention or expectation. There was already a test that checks the RHS $in as a tuple, so I just followed that convention. It doesn’t affect the query behavior.

Jibola · 2025-10-06T23:44:30Z

tests/model_fields_/test_embedded_model.py

+                    "$match": {
+                        "$expr": {
+                            "$eq": [
+                                {"$getField": {"input": "$data", "field": "integer_"}},


This is an example of a value that could get rid of the getField.

Yes, we could get rid of those getField if we use $data.integer_ but I thought it was out of scope for this refactor. This behavior is the current behavior.

Yes, it's out of scope!

Jibola · 2025-10-06T23:47:21Z

tests/model_fields_/test_embedded_model.py

+            [
+                {
+                    "$match": {
+                        "$expr": {


Curious to the callback chain on this one since the null-check could actually be converted.

May best be an improvement added later

django_mongodb_backend/fields/embedded_model.py

tests/model_fields_/test_arrayfield.py

tests/model_fields_/test_embedded_model_array.py

timgraham · 2025-10-13T21:18:17Z

django_mongodb_backend/fields/embedded_model.py

+            if not valid_path_key_name(previous._field.column):
+                return False


The purpose of this could be described with a comment. What's an example of an invalid path key name? And valid_path_key_name could use a comment or docstring. I guess "user.address.city" is a path and "user", "address", etc. are considered "keys"?

A json field could have some rare string as key fields. This test shows the case. Yes I called key the things between dots. 🤔 don't know if there is a better name.

But this is EmbeddedModelTransform... not JSONField's KeyTranform. Is this tested? Tried model_fields_ but all passed with assert False above the return False.

🤔 I thought I added a test. I forgot to add, just we could define a column like:
wifi_column = models.IntegerField(db_column="$$Wifi").
And now I am thinking of the other models and lookups. will they work If the user start to define columns like that?

But maybe this analysis is out of scope of this ticket 😬

Without tests, I'm not sure if there could be other issues. We should ask the team if we want to support it. It seems unlikely based on the discouragements described at https://www.mongodb.com/docs/manual/core/dot-dollar-considerations/#field-names-with-periods-and-dollar-signs.

Test added: test_query_price_column. (should be polished but it shows the problem)

WaVEV force-pushed the lookup-refactor branch 3 times, most recently from 529e0ff to a78f26b Compare September 15, 2025 21:21

timgraham changed the title ~~WIP lookup refactor~~ INTPYTHON-751 Make query generation omit $expr unless required Sep 20, 2025

WaVEV force-pushed the lookup-refactor branch from 8b0c247 to 141f1cf Compare September 23, 2025 03:52

Jibola mentioned this pull request Sep 23, 2025

INTPYTHON-750: Support converting $expr with $getField paths #392

Closed

WaVEV force-pushed the lookup-refactor branch from d11378a to 2c48d11 Compare September 26, 2025 02:26

WaVEV commented Sep 26, 2025

View reviewed changes

django_mongodb_backend/expressions/builtins.py Outdated Show resolved Hide resolved

WaVEV marked this pull request as ready for review September 26, 2025 04:52

WaVEV requested review from Jibola and timgraham and removed request for Jibola September 26, 2025 04:52

timgraham reviewed Sep 27, 2025

View reviewed changes

WaVEV commented Sep 27, 2025

View reviewed changes

tests/model_fields_/test_embedded_model_array.py Show resolved Hide resolved

WaVEV force-pushed the lookup-refactor branch 4 times, most recently from 5827580 to 5b9fa93 Compare October 4, 2025 02:50

WaVEV commented Oct 4, 2025

View reviewed changes

django_mongodb_backend/base.py Outdated Show resolved Hide resolved

WaVEV force-pushed the lookup-refactor branch from 5b9fa93 to b0068d4 Compare October 4, 2025 03:07

WaVEV commented Oct 4, 2025

View reviewed changes

django_mongodb_backend/base.py Show resolved Hide resolved

WaVEV force-pushed the lookup-refactor branch from 7469207 to 9be75aa Compare October 5, 2025 18:43

Jibola requested changes Oct 6, 2025

View reviewed changes

WaVEV force-pushed the lookup-refactor branch from 9be75aa to cd47412 Compare October 6, 2025 21:49

Jibola reviewed Oct 6, 2025

View reviewed changes

WaVEV force-pushed the lookup-refactor branch from 55eae28 to 8a8b441 Compare October 7, 2025 03:48

timgraham mentioned this pull request Oct 9, 2025

Rename DatabaseWrapper operator helpers #417

Merged

timgraham reviewed Oct 10, 2025

View reviewed changes

django_mongodb_backend/fields/embedded_model.py Outdated Show resolved Hide resolved

tests/model_fields_/test_arrayfield.py Outdated Show resolved Hide resolved

tests/model_fields_/test_embedded_model_array.py Outdated Show resolved Hide resolved

timgraham mentioned this pull request Oct 10, 2025

Remove unused **extra_context from Aggregates, Add EmbeddedModelArrayFieldTransform.VIRTUAL_COLUMN_ITERABLE #418

Merged

WaVEV force-pushed the lookup-refactor branch from 966cf73 to fdf7fda Compare October 11, 2025 13:09

timgraham force-pushed the lookup-refactor branch from aea2786 to 175563b Compare October 13, 2025 15:41

WaVEV force-pushed the lookup-refactor branch from fe2cba2 to 6a5665c Compare October 13, 2025 21:56

timgraham reviewed Oct 13, 2025

View reviewed changes

timgraham force-pushed the lookup-refactor branch from e4d5dff to e278c38 Compare October 14, 2025 16:44

WaVEV added 2 commits October 14, 2025 12:45

INTPYTHON-751 Make query generation omit $expr unless required

5c03966

Add unit test.

8d6dd8e

timgraham force-pushed the lookup-refactor branch from e278c38 to 8d6dd8e Compare October 14, 2025 16:45



		def valid_path_key_name(key_name):
		return bool(re.fullmatch(r"[A-Za-z0-9_]+", key_name))

		if not valid_path_key_name(previous._field.column):
		return False

INTPYTHON-751 Make query generation omit $expr unless required #396

Are you sure you want to change the base?

INTPYTHON-751 Make query generation omit $expr unless required #396

Uh oh!

Conversation

WaVEV commented Sep 9, 2025 • edited by timgraham Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Jibola left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WaVEV Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

WaVEV Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

WaVEV commented Sep 9, 2025 •

edited by timgraham

Loading

WaVEV Oct 13, 2025 •

edited

Loading

WaVEV Oct 14, 2025 •

edited

Loading